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linked most strongly with improvement in a school’s writing score on the 
statewide portfolio assessment. An examination of over 1,200 schools with 
complete writing portfolio scores for 1992-96 identified 42 schools with 
consistently improving scores and 18 with consistently declining scores. Site 
visits were conducted at 29 of these schools (22 with improving scores and 7 
with declining scores) . Schools selected for site visits included elementary, 
middle, and high schools; were located primarily in rural areas and small 
towns; and reflected the state’s geographic and socioeconomic makeup as well 
as the three writing-accountable grade levels (4, 8, and 12) . Over 100 
teachers, 200 randomly selected students, and 50 administrators were 
interviewed. Qualitative analyses of the site visit reports yielded 36 
indicators that discriminated between improving and declining schools. 
Statistical analysis revealed that consistently improving schools had 
significantly higher scores on 35 of 36 indicators. The indicators include, 
measures of administrative support, quality of professional development, 
family and community involvement, and various instructional strategies. Each 
of the indicators is described, including narrative guidelines for high, 
medium, and low scores and examples drawn from site visit reports. The value 
of students as informants, the benefits of collaborative research, and 
recommendations for dissemination and replication are discussed. (Contains 13 
references.) (SV) 
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EXECUTIVE SUMMARY 



Introduction 

Since 1990 Kentucky schools have been implementing comprehensive education reform as 
mandated in the Kentucky Education Reform Act (KERA). Among the most successful 
components of this broad reform, in the view of many educators and parents, is the Writing 
Portfolio Program. Overall, Kentucky students write far more than in pre-KERA days and their 
writing has improved, according to writing scores on the state assessment. Yet, performance 
varies widely among schools. In some schools writing performance has improved dramatically, in 
others only casually, and in still others it has remained stable or even declined (Kentucky 
Department of Education, 1996). In 1995 researchers at AEL, Inc., and Kentucky Department of 
Education Writing Program staff, including regional writing consultants, began a five-year 
collaborative research and development project to discover what school conditions and practices 
appeared to be linked most strongly with improvement in a school’s writing score on the 
statewide portfolio assessment and then use that information to help all schools improve student 
performance on writing portfolio development. 

The first phase of this applied R&D project concerned identifying features of writing 
programs and practices that discriminated schools with improving writing portfolio scores from 
schools with declining writing portfolio scores. The objectives of this phase of the study were to 
(1) identify K-12 schools across Kentucky with consistently improving portfolio scores and those 
with consistently declining writing portfolio scores for the years from 1992 through 1996, (2) 
select a representative sample from each group for site visits, (3) develop processes and protocols 
for examining the writing program in the sample schools; (4) conduct site visits to the selected 
schools; (5) identify writing program features, or indicators, that seemed to be common in 
continuously improving schools, and (6) develop a rubric for scoring both improving and 
declining schools on the identified indicators. 



Methodology 

This collaborative project combined qualitative and quantitative research methodologies. 
More than 1,200 K-12 schools in Kentucky had complete writing portfolio scores for 1992-96. 
Of those, 42 schools had consistently improving writing portfolio scores, while 18 had 
consistently declining scores. Schools selected for site visits included elementary, middle, and 
high schools and reflected the state geographic and socioeconomic makeup as well as the three 
writing accountable grade levels (4, 8, and 12). The research team developed and pilot-tested 
interview protocols for use in site visits to the selected schools. These included interview 
protocols for use with students, principals, district administrators, and teachers with diverse 
responsibilities — English/language arts, other content areas, grade levels other than portfolio 
accountable grades, writing leaders in the school/district. 
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After an initial baseline study to determine the feasibility of the site visit protocols and 
processes, teams of two or three members conducted one- or two-day site visits in three main 
time periods of spring 1997, fall 1997, and spring 1998. In the end, 22 continuously improving 
and 7 continuously declining schools were visited. The collaborative research teams interviewed 
more than 1 00 teachers, 200 randomly selected students, and 50 administrators in the 1 8-month 
data collection phase. Site visit reports were written in a common format, based on the 
completed interview protocols. Qualitative analyses of the site visit reports yielded common 
indicators of schools with continuously improving scores. A rubric for scoring each school’s 
report on the indicators was developed utilizing a 1-7 Likert -type scalar form. The scoring work 
included narrative guidelines for low, medium, and high scores on each indicator and also relevant 
examples/illustrations drawn from the site visit reports. 



Findings 

Each school report was scored by two members of the research team; any differences were 
resolved. The consensus score for each of the 36 indicators was entered into an SPSS file for 
analyses. Frequency and distribution analyses were completed for each indicator for each school, 
then for the two main groups of schools — improving and declining. Bar charts were prepared to 
illustrate the large differences in mean scores. T-tests revealed that the consistently improving 
schools had significantly higher (.05 level or less) scores on 35 of the 36 indicators. 

Some of the indicators reflecting schools/district support of the writing program include 
(1) the district demonstrates commitment to the writing program by allocating resources to 
professional development or technical assistance, by assigning program oversight to qualified 
personnel and allocating sufficient time for effective oversight, by compensating cluster leaders 
through additional pay or release time, and in some districts by establishing policies requiring 
portfolio completion for promotion or graduation; (2) the principal actively supports writing 
instruction by providing resources, technical assistance, and/or professional development, etc.; (3) 
a high degree of collegiality is evident among teachers; (4) language arts teachers at the 
accountability grades are confident of their understanding of writing portfolio requirements; (5) 
school writing leaders are satisfied with the level of training and support they have received to 
assist other teachers with portfolio development; and (6) most language arts teachers at the 
nonaccountability levels participate in professional development on writing instruction and 
portfolio development. 

Some of the indicators reflecting instructional strategies include the following: (1) students 
write frequently in all subjects, and the writing is integrated into instruction; (2) teachers in most 
grades and content areas give writing assignments that have the potential of contributing to 
students’ writing portfolios; (3) teachers promote peer conferences as well as student-teacher 
conferencing; (4) teachers spend substantial time on prewriting activities; (5) teachers focus on 
developing writers rather than developing portfolios; (6) teachers provide latitude for students to 
choose topics and/or formats when they write; (7) teachers model parts of the writing process as 
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they work with students; and (8) the mechanics of writing [grammar, spelling, punctuation] are 
taught in the context of writing rather than as unrelated drills and worksheets. 

In schools with continuously improving portfolio scores: students commonly speak of 
themselves as writers rather than students who must complete writing assignments; speak of 
writing as a routine part of their school day rather than as separate tasks done to produce a 
portfolio; expect that writing competence will be necessary in adult life, whatever career path they 
may follow; describe substantive ways their writing has improved from one year to the next, 
including choices of topics, organization, use of supporting details, spelling, grammar, and 
punctuation; express confidence that most students — including themselves — can become 
proficient writers with sufficient effort. 



Conclusions 

Several conclusions are drawn from this project. First, it is possible to identify and 
validate writing program indicators that distinguish schools with continuously improving portfolio 
scores from those with continuously declining scores. Second, including students as informants 
about writing instruction proved invaluable in identifying common practices in the schools. 

Student perceptions about the value of writing, its place in the school, and common writing 
practices helped to validate teacher and administrator perceptions. Third, the collaborative nature 
of the study provided unique benefits. The researchers brought, in addition to research strategies, 
the neutral attitudes and cautions required for scoring well-founded conclusions from the data. 
Regional writing consultants brought deep knowledge of the writing program from their 
experiences as classroom teachers and as people charged with providing technical assistance in 
writing to classroom teachers. Kentucky Writing directors brought awareness of the political and 
regulatory contexts of the program, as well as the ability to gain Department of Education support 
for research activities and access to schools. Last, this project developed guidelines that may 
encourage similar collaboration in the future. 



Recommendations 

Regarding recommendations, the results of this study should be disseminated within 
Kentucky and nationally. These results have implications for school and district administrators, as 
well as classroom teachers. They also have implications for preservice education as well as in- 
service professional development. The protocols and instruments developed in this effort can be 
used by others in replications or for adaptations to other content areas. Due to the strict 
definitions governing the two school groups studied (continuously improving and continuously 
declining over the same four-year period), a similar study should be conducted among the schools 
excluded from the sample — this includes the majority of Kentucky schools. Finally, as one means 
to help satisfy the Kentucky Department of Education’s need for a process by which schools can 
assess their own writing programs, it is recommended that the instruments and processes used in 
this research phase be developed into a writing program self-study, then piloted and field-tested in 
other Kentucky schools. 



INTRODUCTION 



Background on the Kentucky Education Reform Act (KERA) 

Writing portfolios are a critical component of writing instruction in Kentucky, where 
education reform is driven by a high stakes performance-based assessment. The architects of 
KERA, perhaps the most extensive state-level education reform ever initiated, mandated this new 
assessment format in order to encourage changes in teachers’ instructional practices as one means 
of improving education in the state. Although some educators would disagree with the premise 
that assessment drives instruction, KERA is very much based on that premise, as demonstrated 
both by the new assessment design and by the school-level rewards and sanctions that are largely 
determined by student performance on the new tests (Kifer, 1994). Evidence from early studies 
involving several rural Kentucky districts indicates that the reform is indeed impacting instruction 
and that impact is largely due to the design and accountability weight of the assessments (AEL, 
1994). 



Another part of the reform that also aims to impact instructional practices is the increased 
funding of professional development for teachers. The reform designers recognized that changing 
instruction would necessarily go hand-in-hand with intense professional development efforts. In 
order to support this substantial need, legislators increased professional development funds from 
the level of $1 per student at KERA’s inception to the 1999 level of $23 per student. 

Professional development options have not only increased significantly, but many focus directly 
on helping teachers attend to the demands of the new tests (AEL, 1996). Professional 
development in writing is provided largely through the state writing program. The Kentucky 
Writing Program is strongly influenced by the National Writing Project, which began in northern 
California in the mid-1970s. Eight university-based summer institutes using the National Writing 
Project model have been available in Kentucky since before KERA, and other state professional 
development in writing is infused with understandings about the process of writing from that 
model. 



For those unfamiliar with Kentucky’s accountability system, a brief description follows. A 
school’s accountability index is largely determined by student scores on the open-response 
sections of the test and the writing portfolio. Student scores on science, reading, mathematics, 
and social studies, and scores from the writing portfolio section of the test comprise the great 
majority of the accountability index. In 1991-92, the first year of the assessment, the writing 
portfolio formed the entire writing index. Starting in 1996 and continuing, scores from the on- 
demand writing section of the test were included in the writing score, but the on-demand section 
still determines only a small portion of a school’s writing index. Therefore, from the beginning of 
the assessment, the writing portfolio scores have comprised a significant portion of the 
accountability index. 

For this discussion, it is also important to note that rewards and sanctions are determined 
by comparing an individual school’s performance to its own past performance. In order to receive 
rewards, a school must demonstrate sufficient improvement over the past year’s performance. 
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Sanctions in the form of additional technical assistance and dollars occur when schools fail to 
improve sufficiently or demonstrate lowered student performance. 

The writing portfolio assessment has been in place for eight years. Over those years, 
personnel from the Kentucky Department of Education (KDE), especially those involved in the 
Kentucky Writing Program, have noted some obvious patterns in individual school scores. Some 
schools have demonstrated continual improvement in their writing scores over the years, while 
other schools’ writing scores have fluctuated. The current study was initiated in order to identify 
indicators of successful schools’ writing programs. 



Research on Writing Program Indicators 

The research literature offers little guidance in the search for school-level writing program 
indicators. Lipson and Mosenthal, in a paper presented at the 1997 AERA conference, looked at 
indicators by the instructional style and pedagogical stance of individual teachers in relation to 
Vermont’s writing portfolio assessment. They reported that the writing portfolio assessment 
changed how the teachers in their study taught writing, but that the amount and type of change 
depended on the perspective of the teacher. A conclusion of the study, which included surveys, 
observations, and interviews, was that “the diversity of teachers and the variety of their teaching 
contexts must be taken into account” by policymakers attempting to influence practice. (Lipson 
and Mosenthal, 1997, p. 14) 

While the Lipson and Mosenthal study attempted to determine indicators that influence 
individual teacher practice, Rand studies in both Vermont and Kentucky focused on implementation 
at the state level (Koretz, et al., 1992, 1993, and 1996). Teacher surveys revealed that even 
though a significant portion of teachers reported that the assessment was overly burdensome, 
changes in instruction varied widely among teachers. However, none of these studies examined 
school writing programs or attempted to determine indicators that differentiate more successful 
from less successful schoolwide programs. Other researchers have identified state-level indicators 
of enacted curriculum, and of mathematics and science instruction in particular. They offer 
guidelines for determining and using such indicators (Blank, 1993, 1997; Porter, 1991; Smithson, 
1994). The literature is silent, however, on the subject of writing program indicators. An ERIC 
search produced no documents that addressed the topic. 



Project Purpose 

The ultimate purpose of this project is to develop a process that will prepare Kentucky 
teachers to facilitate improved student writing as assessed through writing portfolios. The 
purpose was identified in 1995 by then Commissioner of Education Thomas Boysen, key KDE 
staff, and key AEL staff. Boysen’ s successor, Commissioner Wilmer Cody, continued support for 
the project, designed as a five-year collaborative applied research and development effort. The 
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16-member R&D team, formed in the spring of 1996, includes 12 Kentucky members and four 
AEL members. All continue to be actively engaged in project planning and implementation, even 
though the percentage of time allocated by all personnel is less than 1.5 FTE per year. This paper 
addresses the first project phase: identifying writing program indicators that distinguish schools 
with continuously improving portfolio scores from schools with continuously declining scores. 

The nature and focus of this work was refined in a March 1 996 meeting between key KDE 
R&D and writing program staff and AEL staff. KDE staff were concerned that, as evidenced in 
portfolio scores, some schools’ students were making no progress in writing, and progress in 
other schools was slowing. Department staff felt that early progress might reflect instructional 
changes that were easiest to implement, such as increasing the amount of writing students do or 
the amount of instructional time given to writing. They suspected that continued progress would 
require more substantive programmatic and instructional changes. They wanted to discover what 
writing program features were common among schools that had shown steady progress in writing 
portfolio scores, in order to inform professional development in writing. AEL staff were 
interested in designing and implementing a collaborative applied R&D project with a state 
education agency. Each agency in this collaboration had a vested interest in its outcome. Each 
brought unique information and expertise to the collaboration, the synthesis of which allowed 
insights and developed knowledge neither could have achieved separately. 



Audience 

The primary audience for this report is the collaborative research team of the Kentucky 
Writing Portfolio Project (i.e., the Kentucky Department of Education Writing Program staff, 
Kentucky regional writing consultants, and AEL staff members assigned to the project). 
Secondary audiences are the Kentucky State Caucus of the AEL Board of Directors, which acts 
as an advisory group to this project, and the federal Office of Educational Research and 
Improvement, the AEL funding agency. Other audiences include those interested in school self- 
studies of curricular programs and those interested in school writing programs. 
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METHODOLOGY 



Goals and Objectives 

The first phase of this applied R&D project concerned identifying features of writing 
programs and practices that discriminated improving schools from declining schools, as defined by 
writing portfolio scores. Its objectives were to (1) identify K-12 schools across Kentucky with 
consistently improving writing portfolio scores and those with consistently declining scores for the 
years from 1992 through 1996; (2) select a representative sample from each group for site visits; 
(3) develop processes and protocols for examining the writing program in the sample schools; (4) 
conduct site visits to the selected schools, (5) identify writing program features, or indicators, that 
seemed to be common in continuously improving schools; and (6) develop a rubric for scoring 
both improving and declining schools on the identified indicators. 

The expectation was that the research would lead to professional development products, 
especially a self-assessment tool for school use. The research phase was completed in the spring 
of 1998. The research team identified, through both qualitative and quantitative measures, school 
conditions and practices linked to consistent writing improvement. 



Perspectives and Theoretical Framework 

The methodology used in the research phase of the project combined qualitative and 
quantitative procedures. The conceptual framework for the qualitative methodology was 
grounded theory (Strauss & Corbin, 1990), a cyclical process of moving from theory to inquiry to 
further theory based on the inquiry. The research into portfolio writing was conducted from the 
perspective of the Kentucky Writing Program, which is based in large part on that of the National 
Writing Project. The findings, based on empirical observation and supported through tests of 
interrater reliability, were subjected to frequency and distribution analyses; then t-tests on each 
indicator were conducted. These theoretical and methodological tools resulted in an unusually 
rigorous process. 

A team of eight people, four from AEL and four from KDE, designed the research. The 
research team combined the perspectives of practitioners and researchers both in its design and 
implementation. Initial site visits were made by one AEL researcher and two KDE writing 
program staff members. The first protocols, while based on Kentucky’s Writing Program 
guidelines, were global in nature, in order to capture elements of practice not directly related to 
writing instruction but of possible consequence. Qualitative instruments were developed and 
improved through collaborative work at the conclusion of every round of school visits. 
Researchers learned a great deal about the teaching of writing from practitioners, and 
practitioners learned about the research process as they participated in it. The qualitative 
fieldwork not only generated indicators of effective writing instruction, but also produced copious 
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anecdotal evidence that was used when disseminating findings and illustrating rubrics to assess the 
indicators. A detailed chronology of the methodology follows. 

The first step of the research was to identify schools in Kentucky, at any level, that had 
either consistently improving or consistently declining scores on the writing portfolio portion of 
their school accountability index. The population of Kentucky schools for this identification 
process included all schools with writing portfolio scores for the years of 1992 through 1996. 

Over 1,200 schools had complete data. Consistently improving and consistently declining were 
defined operationally as steady changes in writing scores either up or down, regardless of the 
1992 starting point. That is, one school’s scores might have begun at 50 (of 140), then moved to 
54 in the next year, then 63, and finally 67. Another might have started at 18, then scored in 
subsequent years, 22, 30, and 40. Declining schools were similarly identified. Of the 1,200 
schools, 42 were consistently improving and 18 consistently declining. Ultimately, 22 of the 42 
schools with consistently improving scores and 7 of the 1 8 schools with consistently declining 
scores were visited. The schools selected for site visits included elementary, middle, and high 
schools and reflected the state’s geographic and socioeconomic makeup as well as three 
accountable grade levels (4, 8 and 12). Too, schools were selected to represent various 
enrollment sizes. 

The second step in the project involved the development of processes and protocols for 
collecting detailed writing program information from the identified schools. The research team 
developed and pilot-tested protocols for use in site visits to the targeted schools. These included 
interview protocols for use with students, principals, district administrators charged with writing 
program oversight, and teachers with diverse instructional responsibilities — English/language arts, 
other content areas, grade levels other than portfolio accountable grades, writing leaders in the 
school and/or district. Interview protocol questions ranged from specific classroom activities to 
broad program issues such as level of professional development in writing instruction and 
program funding. A data collection protocol elicited information about resources available for 
writing instruction, both material and human. These initial protocols went through many 
refinements as they were employed in the various stages of school visits. This process of 
discussing interviewees’ responses and honing questions for subsequent school visits produced the 
indicators of successful school writing programs. 

After an initial baseline study to determine the feasibility of the site visit protocols and 
processes, teams of two or three members, including researchers and practitioners, first visited 
selected schools with continuously improving scores, and subsequently visited schools with 
declining scores. These one- to two-day visits occurred in three main time periods of spring 
1997, fall 1997, and spring 1998. The visits included data collection and interviews with people 
from the role groups identified earlier. 

Names of students to be interviewed were drawn at random and stratified by grade levels. 
This stratified random selection was weighted toward students in portfolio accountable grades. 
Students were interviewed in groups of two or three, although their responses were recorded 
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individually. All adults were interviewed individually. More than 100 teachers, 200 students, and 
50 administrators were interviewed in the 1 8-month data collection period. After each school 
visit, the research team developed a summary report from an analysis of the interviews and 
resource inventory. These reports, organized by the refined questions, were the bases for 
developing the indicators of successful school writing programs. 

The final steps of this process were to devise a rubric by which schools could be assessed 
on each indicator, complete the assessments, and compute the group mean score on each indicator 
for the continuously improving and continuously declining sets of schools. A seven-point Likert 
scale (1 = low, 7 = high) was used to rate each school on each indicator. The scoring rubric 
included narrative guidelines for low, medium, and high scores for each indicator. Also, accounts 
of behaviors or conditions found in school visits were interspersed in the narrative for selected 
potential indicators. These illustrations were collected purposely during the site visits. Thirty-six 
writing program indicators were identified at this point. 

Each school report was individually scored by at least two members of the research team. 
Scorers then met to resolve any scoring differences. Those reports that contained indicator 
ratings that could not be resolved were scored by a third team member. The consensus score for 
each indicator from each school was entered into an SPSS data file for analysis. Frequency and 
distribution analyses were completed for each indicator for each school, then for the two main 
groups of schools — improving or declining. Bar charts of each indicator by school and by the 
means of the two school groups were prepared for the research team. Lastly, for each indicator t- 
tests were conducted on the differences between mean scores for the improving versus declining 
groups of schools, and the results were summarized. 
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FINDINGS 



The findings are presented in two sections. The first section presents the qualitative 
research results. It identifies the indicators that distinguished continuously improving from 
continuously declining schools and presents descriptions of high, middle, and low performance for 
each indicator. It also includes brief accounts of situations or activities seen during site visits that 
illustrate performance levels on some of the indicators. The second section displays the indicator 
results in descriptive and inferential statistics format. The reader will note that in the second 
section the indicators are identified by their computer program variable names and shortened 
titles. 



Qualitative Findings 

The qualitative research in continuously improving and continuously declining schools 
yielded a set of indicators that appeared to distinguish the more successful from the less successful 
school writing programs. All but one of these indicators was later strongly verified through 
statistical analysis, and that indicator is also reported below, because the lack of statistical 
significance may easily be an artifact of the small number of schools visited. 

Indicators. Following is a brief description of each indicator (describing the highest level 
of implementation observed during school visits). 

1. Administrative support 

A. District level support 

Administrative support includes any formal or informal evidence that the district 
administration gives the writing program high priority in the district curriculum. Such 
support may look different from district to district depending on the resources of the 
district, the number of central office staff, and local history and culture. The key criterion 
is that, for this district, consistent effort is going into proactive encouragement of the 
writing program. 

High — The district gives verbal support to the writing program and institutes district 
policies intended to stimulate writing improvement (e g., requiring a completed portfolio 
at a specified minimum performance level for high school graduation); encourages and 
supports school initiatives to improve writing (e.g., providing funds for professional 
development above that expected by the state, providing release time for mentoring); 
recruits and/or trains expert, facilitative central office staff to provide technical assistance 
and mentor school staff; organizes portfolio scoring for the district to ensure accuracy; 
organizes cross-school or cross-grade conversations about writing instruction. 
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Example: A western Kentucky district provides funds so that groups 
of teachers can attend conferences or visit other schools. The district 
also facilitates meetings and work sessions for teachers from all 
schools to work together. This practice reduces interschool 
competition and encourages sharing ideas. The principal of one small 
elementary school and the district have organized a resource center 
containing professional journals and other resources. The center is 
open to all teachers in the district until 6:00 p.m. daily during the 
school year. 

Other types of district support were observed more rarely but seemed very effective (for 
instance, cross-school meetings to align curriculum and to score at the district rather than 
the school level). These meetings provide extensive technical assistance to writing 
teachers. Some districts have created a writing resource teacher position. This individual 
organizes scoring sessions and provides technical assistance throughout the year. 

Medium — The district gives verbal support to the writing program, providing assistance 
that is perceived by the school staff as more facilitative than is provided in other -r 
curriculum areas (for instance, encouraging schools to provide more training for teachers 
than is generally expected but requiring them to use school professional development 
funds for the purpose), identifying some of the more successful practitioners to facilitate 
the writing program at the district level. 

Low — The district does only the minimum required by the state, assigning portfolio 
oversight to personnel who lack clear qualifications or successful practice, maintaining 
bureaucratic oversight of schools' writing programs with little or no technical assistance. 

Example of a medium to low district we have visited. The district 
had a cluster leader who served as mentor to the writing leader in the 
school but subsequently reassigned that staff member to a different 
position that prevents her from visiting schools and mentoring writing 
leaders. The district encourages schools to use their professional 
development funds to send the writing leader and intern teachers to 
intensive professional development workshops not available to most 
teachers but does not make any district funds available for the 
purpose. No stipends are provided, but expenses are paid — in sharp 
distinction to what is provided for “extra” professional development 
in other areas. 

B. School level support 

Administrative support includes any formal or informal evidence that the school 
administration gives the writing program high priority in the school curriculum. 
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High — School administrators provide active leadership; change schedules to allow 
additional time for planning, conferencing or dialogue among teachers or time for 
instructional aides or volunteers to conference with students; provide extra time and/or 
payment for lead teachers; encourage generous allocations for professional development 
and instructional resources in writing in the school budget and find additional funds when 
needed to take advantage of unanticipated opportunities; participate in professional 
development with teachers; serve as readers for student writing, participating in portfolio 
scoring, initiating efforts to bring student work to real world audiences. 

Examples from schools we visited: To illustrate that active principal 
leadership need not always be hands-on, we visited a school where the 
principal did not personally participate in the writing program but 
encouraged teachers at all grade levels to take as much training as 
possible in the writing process and in portfolio development. In addition, 
he persuaded the school council to allocate a full-time aide to the fourth 
grade, to allow extra planning time for fourth-grade teachers and an 
additional adult to conference with students about their writing. He 
delegated full responsibility for the writing program to the fourth-grade 
teachers, who were regarded as the school’s writing leaders, but he made 
sure they had more resources than teachers at other grade levels (for 
instance, were able to take more field trips if they felt it was necessary for 
the writing program) and celebrated their creative ideas to stimulate 
student writing and to share that writing with real-world audiences for 
purposes that were important to the students. 

Administrative support for the writing program can take forms other 
than books, equipment, and dollars, as reported by an eastern Kentucky 
middle school. To help “reluctant writers,” teachers contact parents and 
make every effort to encourage the students to complete their writing 
assignments. If these efforts do not work, the principal schedules a 
principal/teacher/student conference. The principal leads the 
conference, after reviewing background information from the teacher. If 
this conference is unsuccessful, the principal assigns the student to 
“Saturday School” to complete the missing writing pieces and sends a 
message home to the family about the required attendance. This four- 
level process usually results in completed work. Teachers say they 
appreciate this level of administrative support for the writing program. 

At a K-4 school in central Kentucky, fourth-grade teachers get support 
for writing portfolio development activities in their classrooms. With 
the encouragement of the principal, the faculty voted to hire an 
instructional assistant for those teachers, using the school’s instructional 
funds. Primary teachers are willing to forego some of the resources they 
might otherwise have in order to support the teachers who are “on the 
line.” 
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Medium — School administrators make sure that teachers are provided with up-to-date 
information, professional development, and hands-on technical assistance from competent 
district staff; provide release time or stipends for extra duties associated with the writing 
program, such as portfolio scoring; allow and encourage teachers to participate in 
professional development over and above that called for in the school’s PD plan; but let 
the central office take the lead in encouraging the writing program and do not change the 
school’s staffing or organization to improve the writing program. 

Example: We visited a high school where the principal had no direct 
involvement in the writing program but rather delegated responsibility 
for encouraging the program to an assistant principal and the cluster 
leader. The cluster leader took the initiative to approach the KDE 
Writing Program with questions about whether particular types of 
real-world writing were appropriate to include in the portfolio. She 
was able to involve non-language arts teachers in the writing program 
to an unusual degree, but primarily on the basis of her own reputation 
in the school, not administrative support. 

Low — School administrators obey state directives at a minimum level, perhaps monitoring 
implementation of the writing program without actively assisting it, failing to encourage 
professional development in writing for most teachers, and providing no special resources 
to enhance writing instruction or portfolio development. 

2. Availability and use of quality professional development/technical assistance 

This indicator refers to the entire writing program, not just portfolio development and/or 
scoring. It considers both formal and informal opportunities. Formal professional 
development denotes planned experiences led by one or more experts in the field. Informal 
professional development denotes mentoring relationships, coaching, electronic networks, 
peer dialogues about writing instruction sustained over time, technical assistance provided on 
request to individual teachers. The first three indicators below require a global assessment of 
the quality and quantity of professional development (in the broad sense) received by teachers 
at the school. The remaining indicators under this heading require judgments about specific 
types of training or informal professional development opportunities of particular interest to 
this project. 

A. Professional Development at the school level for writing leaders (cluster leaders or 
other teachers assigned responsibility for the writing program) 

High — The writing leader has — with the principal’s support in the form of stipends, 
release time, and/or other substitutes — attended professional development in writing 
beyond what is offered to others. In turn the leader provides workshops to school 
colleagues, and has time to coach and mentor other teachers. The leader has in-school 
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access to and reads professional journals and books on writing instruction, and shares 
information with colleagues. 

Medium — The leader is allowed to get professional development beyond what is available 
to other teachers, but is only occasionally provided support for participation. He or she 
provides resources to colleagues as she or he discovers them, but has no reliable source of 
information. The leader may meet briefly once or twice a year with other teachers to 
provide updates on the writing program or demonstrate a strategy, but doesn’t have time 
to offer ongoing assistance. 

Low — The writing leader’s primary role is as a conduit of information and a portfolio 
manager. He or she is not expected to provide assistance or training to other teachers or 
to seek professional development beyond what is offered in the school or district. 

B. Professional development at the district level for writing leaders (cluster leaders or 
others teachers assigned responsibility for the writing program) 

High — Writing leaders are encouraged to participate in much more than the minimum 
professional development on writing expected of those in their positions. They seek out 
appropriate conferences and workshops to attend; have a support network (within the 
district, with peers in other districts, with consultants, with KDE Writing program staff); 
they have district on-demand technical assistance and/or coaching. In turn they provide 
training and support to their colleagues beyond what is required. 

Medium — Writing teachers are encouraged to seek out appropriate conferences, 
workshops, and technical assistance but are unable to avail themselves of all appropriate 
opportunities because of district limitations (e g., financial limits, lack of substitutes) or 
personal limitations (e g., a demanding family situation). Opportunities to share what they 
have learned with teachers in the school may also be limited, for some of the same 
reasons. They have a limited network to call on. 

Example: In one school a teacher was the designated writing program 
leader. He was permitted to attend conferences and workshops but had 
limited knowledge of what was available. He was unaware of a summer 
writing program workshop at a nearby university, for instance. Not being a 
cluster leader, he did not have access to the resources provided through the 
state department of education. He had structured the school’s writing 
program around writing instruction in his own field (Title 1), and he clearly 
understood the writing process better than his colleagues. 

Low — Writing leaders have attended the minimum amount of professional development 
on writing expected of people with their responsibilities. They have either not sought or 
have not had the district level support necessary to participate in additional training. They 
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have formed no support networks. They have distributed district information about 
writing to other teachers in the school, but have offered no other assistance. 

C. Overall professional development for portfolio accountable teachers (teachers 
responsible for teaching writing at the accountable grade) 

High — Teachers responsible for writing at the accountable grade level have participated in 
much more than the minimum professional development on writing expected of those in 
their positions (including training in portfolio scoring). They attend conferences, have a 
support network, and are able to use on-demand technical assistance and/or coaching. 
They feel some responsibility to let other teachers who are less accountable for student 
portfolios know how to work with students to improve portfolios. 

Medium — Only some teachers of writing at the accountable grade level have received 
extensive training; others have had professional development in the writing process and in 
portfolio scoring but tend to rely on a few leaders who have developed a support network 
to rely on, and who are willing to support the less involved teachers. 

Low — Writing teachers have attended the minimum amount of professional development 
on writing expected of people with their responsibilities (if they are cluster leaders) or no 
more professional development than most other Kentucky writing teachers (if they are not 
cluster leaders). They have either not sought or have not had the support necessary to 
participate in additional training. They have formed no support networks. They have 
provided minimum support to other teachers in the school. 

D. Overall professional development for other teachers (teachers in subject areas other 
than language arts and language arts teachers in nonaccountable grades) 

High — Teachers have participated in much more than the minimum professional 
development on writing expected of those in their positions (often because the school or 
district has sponsored extensive in-service professional development in the writing 
process). They attend conferences, support one another routinely, and have resources for 
on-demand technical assistance and/or coaching that they find helpful. 

Example: The primary teachers at an elementary school we visited 
worked together to design and develop a cumulative portfolio, starting 
with a few pieces in the first year of the primary program and adding 
pieces each year, so that students entering fourth grade had full working 
portfolios. The fourth-grade teachers worked closely with the primary 
teachers to develop these portfolios. Students reported that assembling 
their final primary portfolio was a more rigorous learning experience than 
putting together the fourth-grade portfolio. 
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Medium — Most teachers interviewed have participated in workshops or other 
professional development on the writing process and on scoring portfolios, but content 
area teachers at other than the accountable grade level make little use of the training. 
Instead, they depend on writing teachers at the accountable grade to take responsibility for 
the writing program and for portfolios. Networking is limited to writing teachers talking 
with other teachers about what they can do to strengthen students’ writing skills and their 
portfolios. 

Example: At most of the departmentalized schools we visited, writing 
in the content areas was the most problematic part of the writing 
program, even when the content area teachers were convinced that 
writing was important to their disciplines. Finding appropriate 
professional development opportunities was often difficult. In a high 
school where the great majority of the content area teachers 
cooperated willingly with the English department in writing 
instruction, the English teachers asked that the content area teachers 
follow the writing process only through the first draft stage. If the 
piece was considered appropriate for a portfolio, it was sent to the 
English department teachers to help the student edit and publish the 
piece for the portfolio. At other high schools, language arts teachers 
complained that content area teachers would send them writing pieces 
that were not good enough to be included in a portfolio or that were 
based on inadequate prompts. 

Low — Teachers have attended the minimum amount of professional development on 
writing expected of teachers in their content area or grade. They either have not sought or 
have not had the support necessary to participate in additional training. They have formed 
no support networks. They have provided minimum support to other teachers in the 
school. 

E. Attention to writing that is appropriate for content areas other than language arts 

High— Recent professional development opportunities for all teachers have featured ways 
to incorporate writing authentically into content areas other than language arts, and the 
school writing leader or leaders have followed up with continued technical assistance to 
help teachers use the instruction. In addition, teachers in content areas other than 
language arts may have been encouraged to take intensive instruction in writing for their 
content areas and have subsequently shared what they have learned with others in the 
school. 



Example: In a mid-sized high school, the language arts faculty serves 
as a cadre of teachers and mentors for the entire faculty in writing 
portfolio development. These teachers receive an extra planning 
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period, so that they can have time to work in small groups or 
individually with teachers in other content areas. They help prepare 
open-ended questions and develop rubrics to enhance writing across 
the curriculum. Each content area is required to submit two portfolio- 
ready pieces of writing per year. Building in-house capacity has 
benefited the faculty as well as the students. 

Medium — Some but not all teachers have had the opportunity for professional 
development emphasizing how to incorporate writing into content areas other than 
language arts. These opportunities may have been made available only to non-language 
arts teachers who have expressed a particular interest in using writing to enhance their 
instruction. 

Low — There is no evidence that attention to writing that is appropriate for content areas 
other than language arts has been identified as a school need, and no one appears to be 
seeking professional development in this area. 

F. Teacher training (and participation) in portfolio scoring 

High — All teachers in the school are trained annually and participate in portfolio scoring 
(at either the accountable grade level or other grade levels where completed portfolios are 
required by the school or district). (If teachers are scoring portfolios at non-accountable 
grade levels, they may be trained by a writing leader other than the cluster leader or 
district assessment coordinator.) 

Example: One of the schools we visited has a scoring team 
consisting of all the fourth-grade teachers. Students at all grade 
levels (primary and upper elementary) are required to keep portfolios, 
and each grade-level team of teachers scores its own students' 
portfolios. Teachers at the non-accountable grade levels are trained 
by the grade-level scoring team, after it has had updated training each 
spring. 

Medium — Portfolio scoring training is available to all teachers in the school but is 
voluntary for teachers who are not on the scoring team. The portfolio team includes 
teachers from subject matters other than language arts and/or from non-accountable grade 
levels, and the team’s membership changes fairly frequently, to encourage participation by 
any teacher who is interested in participating. 

Low — Only teachers who are on the scoring team receive training in portfolio scoring; the 
team consists only of those who teach writing at the accountable grade level, and the 
composition of the team has changed minimally over the years (frequently only as teachers 
move from one grade level to another or retire). 
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G. Ongoing mentoring and informal professional development 

High — Teachers are members of formal teams (subject-matter teams, grade-level teams, 
primary “families,” special purpose teams) that have adequate time for meeting and 
planning. In addition, networks with teachers outside the school are encouraged through 
sending teachers to appropriate professional development opportunities at regular intervals 
(e g., professional association annual meetings, summer institutes, and the like), providing 
e-mail access for electronic networking, providing release time if needed to network with 
peers outside the school, etc. Informal networking and mentoring within the school are 
also encouraged and celebrated. 

Medium — Teams of teachers within the school are given common planning time, but the 
teachers report either that it is inadequate for substantive work or that they do not make 
consistent use of it. Networking with teachers outside the school is possible if a teacher 
persists in following up opportunities, but is not a school priority. Informal mentoring and 
networking within the school are tolerated but not encouraged. 

Low — There is minimal communication between teachers and administrators, mostly 
formal communications concerning the school. Teachers seldom talk among themselves 
about issues of teaching and learning, and any discussion of students is at a superficial 
level. 

H. Level of strategic collaboration in professional development 

This indicator refers to degree to which the school faculty participate together and support 
one another in professional development experiences linked with identified school needs. 

High — The school faculty has participated together and supports one another in 
professional development experiences linked with identified school needs. Teachers show 
awareness of the school's professional development plan and its connection with school 
weaknesses; most have had a part in developing the plan. Professional development is 
often school-wide. Teachers who attend conferences or workshops beyond school-based 
professional development attend as representatives of their school and are expected to 
bring learnings back to their colleagues. Teachers describe instances of giving and 
receiving help from one another. 

Medium — Teachers are aware of the professional development plan but most were not 
involved in developing the plan and they appear to know the specifics only so far as they 
are directly affected. Collegiality is encouraged, and there is informal collaboration (self- 
selected groups who share), but these groups appear to be mutually exclusive and not 
everyone participates. There is no formal mechanism — or encouragement — for teachers 
to support one another in professional development experiences. 
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Low — Decisions about professional development are commonly made individually, and 
teachers may be unaware of the specifics of the professional development plan developed 
at the school. When teachers are trained together, they are in role-alike and/or grade-alike 
groups. Teachers are generally unaware of their colleagues’ level of training or expertise, 
and they are not encouraged or required to share what they learn in workshops with other 
teachers. They do not mention instances where teachers have helped one another and they 
do not appear to link identified school needs with professional development. 

3. Coordination across grade and subject areas 

Since writing is an area in which students should be making steady, ongoing improvement, 
coordination of the writing program across grade and subject areas is an asset to the program 
(but was not found consistently even in schools with steadily improving writing scores). 

High — In departmentalized schools, teachers from different subject area departments 
share in the same professional development on writing and compare notes on a regular 
basis to ensure that the writing program is implemented in a planned, consistent manner to 
produce integrated, well designed writing instruction. In all schools, teachers from 
adjacent grade levels compare notes on a regular basis to ensure that writing instruction is 
consistent as students progress from grade to grade and that student progress in writing is 
well documented as they move from grade to grade (for instance, through the 
development of a cumulative portfolio). Feeder schools provide the schools that students 
move to with adequate information about each student’s progress in writing and meet to 
ensure that the writing program at the higher level school builds on the program at the 
feeder school. 

Medium — In departmentalized schools, teachers from different subject area departments 
are encouraged but not required to participate in the same professional development on 
writing and to confer with one another periodically about the way writing is used in each 
subject area. Some teachers do this, but others do not. Similarly, teachers at adjacent 
grade levels may or may not coordinate their writing programs; usually some effort is 
made to ensure that student writing moves with the student from grade to grade, but the 
effort is not always successful. Teachers at the accountable grade levels are usually the 
most conscientious about coordination with other teachers. 

Low — No effort is made to coordinate writing instruction across subject areas or across 
grade levels. Individual teachers may express frustration at the lack of coordination. 

4. School climate/communication 

This indicator includes any evidence that relationships among administrators, teachers, 
students, and parents are friendly, collegial, and facilitate the development of the writing 
program. 
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High — Frequent, relaxed and cordial communication occurs among administrators, 
teachers, aides, and support staff; there is evidence of mutual respect and liking among 
faculty, staff and students, as well as any parents who may be present. School displays 
and conversation show that writing is valued as a worthwhile and enjoyable activity. 

Teams of teachers who clearly work well together are another sign of a positive school 
climate. 

Example. An elementary school we visited has a large number of 
students whose parents transport them to school and some students 
were consistently tardy. Now the school starts every morning with an 
assembly to get the day off to a happy, enthusiastic start. Teachers 
are responsible for planning the assembly, which generally features 
either a speaker from the local community or students who have done 
something worth exhibiting. If students are late, it does not disrupt 
instruction, but students are now pressuring their parents to get them to 
school on time because they enjoy the assembly, and parents 
occasionally stay to attend the assembly with their children. The school 
has an extremely diverse student population, both ethnically and 
economically, but the administrators and staff reach out to all students 
and all parents in ways that appear to make all feel welcome and included. 

Medium — Cordial but businesslike relationships exist among administrators, teachers, 
staff, and support aides, with little emphasis on teamwork or meeting the needs of the 
“whole child”; there may be an emphasis on academic excellence but not on teamwork to 
produce academic excellence. 

Low — Hallway, lunchroom, and teachers’ lounge conversations are generally unrelated to 
instruction; there is minimal and formal communication between teachers and 
administrators, with support staff always clearly subordinate to teachers; there is an 
emphasis on order (and usually silence) in the hallways and in the classrooms, with 
exchanges between adults and students in non-classroom settings primarily directive rather 
than social. Hall and classroom displays appear to be mostly adult-created (or “cookie- 
cutter” student work), with little apparent value given to student writing. 

5. Communication with families and family support of the writing program 

High — The school recruits and trains volunteers to help in. the classroom (for example, to 
free teachers for conferencing, to read and give feedback on student writing, to share their 
own writing, to talk about the kind of writing necessary in their jobs). In “family night” 
sessions, through newsletters, or through formal training, the school provides information 
to families about the writing levels expected of students and ways parents can help at 
home. A conscious effort is made to help parents feel comfortable in the school. 
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Example: An elementary school we visited provides classes for parents 
through the library program. The first class last fall gave instruction in 
the writing process and appropriate ways for parents to work with 
children at home on writing. At the same school, one of the children we 
interviewed described how she herself had instructed her mother in 
appropriate conferencing techniques. 

At another school, a high school senior reported, “I conference with my 
mom because she opens the key to my mind.” 

Medium — The school provides written information to parents about the writing process 
and suggests ways they can help students at home with writing assignments, but does not 
actively recruit parents to help in the classroom. Individual teachers may encourage 
parents to volunteer in the classroom or give them pointers on how to work with children 
at home on writing. 

Low — Any communication to parents about the writing process stresses what they should 
not do (e.g., make suggestions that usurp the child's ownership of the writing) rather than 
active ways they can help. Parents are not encouraged to volunteer in the classrooms, 
although a few active parent volunteers may be tolerated. No effort beyond required 
reports is made to explain to parents how a child's writing skills are progressing. 

6. The school uses community resources in the writing program 

High — The writing leader actively seeks resources for such programs as having a writer 
visit the school for a period and work with students on writing as well as explaining how 
he/she makes a living as a writer. Individual teachers are encouraged to invite into the 
classroom people in the community who use writing in interesting ways in their adult 
careers. Such people may also be asked to judge school writing competitions or projects. 
Community firms or businesses may be asked to support the school's writing program in 
various ways (for instance, exhibiting student writing or providing prizes for a contest). 
Relatively few schools were ranked high on this indicator. 

Example: Each year at a small town elementary school the fourth-grade 
students organize a blood drive for the local health department. They 
make all the arrangements for a blood drive at the school, and then they 
write persuasive pieces to convince members of the community to 
participate. Before the drive, they parade through their small town 
leaving these persuasive pieces at each store, to be distributed to the 
patrons. They also talk to their parents and neighbors about the blood 
drive. The blood drive requires expository writing that describes how to 
organize a blood drive, and discusses research on blood types, the need 
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for blood, and how the blood will be used. Some students write 
personal narratives or fiction based on the research they have done. 

Medium — Individual teachers seek out community resources to enrich the writing 
program from time to time, but it is not a school priority. 

Example: The writing leader at a small, rural elementary school invited 
the school superintendent to visit the school and talk to the students 
about the many ways in which writing had played a part in his career. 

She wanted to find some way to support a writer in residence program 
but, to date, the visit by the superintendent was the only effort made at 
the school to use community resources to enhance the writing program. 

Low — There is no evidence that community resources are used in the school's writing 
program. 



School/Classroom Strategies 



7. Focus and intensity of writing instruction programs 

This indicator requires a global assessment of the degree to which writing and/or writing 

portfolios are a clear school priority. 

High — There is abundant evidence that the school communicates to students the 
importance attached to writing. Student writing is prominently displayed in the hallways 
and in classrooms, and classroom walls display posters emphasizing various aspects of 
writing instruction/portfolio development. Interviews with administrators, teachers, and 
students are consistent in revealing the importance placed on writing at the school — and 
interviewees agree that writing well is an important life skill. If events outside the control 
of the school administrators interfere with instruction, an effort is made to ensure that it 
does not interfere with students’ opportunity to work on writing. 

Medium — Writing is a clear school priority, but there is a lack of focus in the writing 
program. Students do not articulate the importance of writing as clearly as teachers and 
administrators do, or they may report that teachers are not actually using all the strategies 
in teaching writing that they report to the interviewer. There is a clearer focus on 
portfolio development than on writing as a valuable part of education, and writing may be 
a focus only part of the year. Teachers may draw a clear distinction between writing 
assignments meant for the portfolio and other writing assignments. Teachers are 
ambivalent about the value of writing, on the one hand considering it valuable but on the 
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other hand resenting the time it takes away from other subject matter, since writing is not 
well integrated into instruction. 

Low — Writing appears to be stressed only at the accountable grade levels and only for the 
purposes of developing writing portfolios as part of the state assessment program. 
Teachers and students resent the demand to assemble writing portfolios and many dislike 
writing. 

8. Use of The Writing Portfolio Teacher's Handbook as a resource by all teachers teaching 
writing 

This indicator is strongly related to the overall priority placed on writing at the school and the 
leadership abilities of cluster leaders and other writing leaders in the school. If a school uses a 
district-produced Handbook rather than that produced by the state Writing Project, it should 
be examined or described well enough so that a judgment can be made whether it is or is not 
equivalent. 

High — Every teacher in the school receives the Handbook and guidance in using it as part 
of professional development, and the writing leader(s) in the school continue to work with 
teachers to demonstrate productive uses for the Handbook. Teachers at all grade levels 
and in all subjects report using the Handbook consistently, though they may turn to the 
Handbook only during certain parts of the year (for instance, when introducing students to 
the benchmarks for the various performance levels or when preparing to score portfolios). 
If they do not turn to the Handbook frequently, they report that they are familiar enough 
with the content so that they do not need it for general reference purposes any more. 
Students exhibit familiarity with the benchmarks and the criteria for the various 
performance levels. 

Medium — All portfolio-accountable teachers have copies of the Handbook and use it as 
described above, and a few other teachers who have a particular interest in the writing 
process may have a copy. Other teachers have guides to teaching writing that are not as 
extensive as the Handbook (often produced by a local writing leader) or have gained their 
knowledge of portfolio development entirely from professional development workshops. 
Some teachers are not aware that the full Handbook exists. 

Low — Only the teachers who are on the portfolio scoring team have copies of the 
Handbook, and they refer to it only when preparing to score portfolios. 

9. Teachers and students focus on writing 

A. Evidence that writing is the focus of the writing program, not just portfolios 

It is to be expected that any school in Kentucky will focus on completing writing 
portfolios. There should also be a focus on writing for its own sake or as an important 
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part of instruction. A key criterion here is the degree to which portfolio writing is 
integrated into classroom instruction or is treated as separate from classroom instruction 
(inevitably taking time away from other instruction). 

High — Writing is part of instruction at all grade levels and in all subject areas. Writing to 
learn and writing to exhibit learning are stressed as valuable, in addition to portfolio- 
appropriate writing. Writing assignments are not differentiated into portfolio assignments 
and other assignments, and potential portfolio pieces may come from a wide variety of 
sources, including writing the student has done for non-school-related purposes. Students 
are exposed to a variety of ways in which writing is needed for adult life and in vocational 
areas. Teachers are conscious of developing their own writing skills and of the 
importance of students’ enjoying writing. Both teachers and students can describe their 
progress as writers. 

Medium — Writing is part of instruction at all grade levels but some subject matter 
teachers resent any demand that they incorporate writing into their instruction and refuse 
to do so. There is more emphasis on writing in the accountable grades and during the time 
that portfolios are being prepared than at other times, although students do some writing 
throughout the school year and in every grade. Although students speak of themselves as 
writers and often write when they are not required to for school assignments, teachers 
generally do not write except when they need to for clearly defined purposes. 

Low — Writing is neglected in the school curriculum except for the preparation of writing 
portfolios — and sometimes for practice in answering open response questions. Writing 
assignments are clearly identified as portfolio assignments and are not integrated into 
instruction. 

B. Teachers require more challenging work from students than they once did because 
students come better prepared 

High — When asked to account for the improvements in student writing, most teachers 
mention that writing has been well taught in previous grades so they are able to work with 
students at a higher level than previously. Both teachers and students describe particular 
assignments that require more than students previously would have been asked to do at 
that grade level. 

Medium — Teachers report that student writing has improved over the years and that they 
now require more of students than they used to, but add that some students have always 
had the capacity to do the work now required and that some students come to them still 
lacking the skills to do the work they now require. 

Low — Teachers report that teachers at previous grade levels have not done a good job of 
teaching writing and that this hampers portfolio development because they must spend 
time working on basics that the students should have learned previously. 
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C. Evidence of students’ confidence that they can eventually reach proficiency 

High — All students interviewed at the school report that the portfolio standards are not 
too high for all (or most) students to reach — though they require hard work from most 
students. Students may add that not all students are willing to put forth the effort 
required, but they consider themselves good writers and report that they are making the 
effort personally. Teachers and administrators agree that most students — over 
time — should be able to reach the standards and feel responsible for helping them do this. 
Teachers do not justify students’ failure to reach proficiency by lack of innate talent or 
family traits. 

Medium — All or most of the students interviewed at the school report that the portfolio 
standards are not too high for all (or most) students to reach — though they require hard 
work from most students. Students may add that not all students are willing to put forth 
the effort required, but they consider themselves good writers and report that they are 
making the effort personally. Most teachers have reservations about the ability of some 
students to become proficient, either because they lack the ability (some have writing 
talent and some don’t) or because they lack motivation or an adequate home environment. 
(Teachers may give other, less self-serving reasons for their reservations: e g., the schools 
will never have adequate support or time to bring all students to the proficient level; or 
they may criticize the performance level itself, e.g., “proficient” is developmentally 
inappropriate at their grade level.) 

Low — Students express doubt that they will ever become proficient or say that a number 
of other students will not be able to do so. Teachers have major reservations of the sort 
described under “Medium.” 

D. Students speak of themselves as writers, not just as completers of writing 
assignments 

Students feel an ownership of their writing and use writing for purposes they feel are 
important. When students speak of completing assignments or talk about what they “had 
to do” for the teacher or to complete a portfolio, they imply that they see themselves as 
obliged to complete work for a grade rather than as having personal motivations to write. 
Students who describe work in which they take pride, who talk about showing their work 
to others (family and friends as well as classmates), who want reactions from people other 
than teachers, or who submit their work for publication outside the school are speaking as 
writers. Students who give examples of writing they do that are not required for class 
assignments (joumals/diaries kept at home; poetry for friends and family; reports or letters 
for clubs, churches or other extracurricular activities, for example) also are speaking as 
writers. 
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High — Most of the students interviewed speak as writers rather than simply as completers 
of writing assignments. Student ownership of writing appears to be the school norm. 

Medium — Some students (half or fewer) speak of themselves as writers, but others do 
not. It may be clear that students of some teachers have ownership of their writing, but 
students of other teachers consistently do not speak as writers. 

Low — Few students speak as writers. Those who do appear to belong to an elite student 
group who consider themselves to be or are considered unusually talented writers. 

E. Students are able to cogently compare their current writing with writing done 
earlier, even in previous years 

The opportunity to compare current with past writing increases students’ sense of 
ownership and pride in their work. It allows them to rely on their own judgment about 
how they have improved rather than relying on external measures, such as grades. Some 
students may not be able to make such judgments without access to a “working portfolio” 
that follows the student from grade to grade. But when a school makes explicit provision 
for students to compare their current work with their past work, the chances that most 
students will increase their ability to critique their own work are greatly enhanced. 

High — All or most students describe substantive improvements in their writing from the 
beginning of the year or from one year to the next (such as improved organization, use of 
supporting details, and style — word choice, better sentence structure). In most cases 
where this is the school norm, the school assists students by routinely collecting writing 
the students value in “working portfolios” that may include work from a variety of content 
areas, formats, and stages of development. The “working portfolio” travels with the 
student from grade to grade and becomes a resource when the student needs to select 
pieces to develop for the assessment portfolio. Presence or absence of a “working 
portfolio” is a strong indicator of students’ ability to compare current with past writing, 
but should be looked at only in conjunction with students’ ability to describe their growth, 
since not all “working portfolios” are used in equally helpful ways and some students may 
keep their own work together, whether or not the school assists. 

Medium — When students speak of the improvement in their writing, most refer primarily 
to superficial improvements, such as improved spelling and mechanics or writing longer 
papers. Even if students keep their writing in a “working portfolio,” it may not travel with 
the student from grade to grade, or the student may not have easy access to it. 

Low — Most students don’t know whether they’ve improved or say that their writing has 
declined (perhaps for lack of practice), or are unable to support a feeling that they have 
improved with specific details. Portfolios are used only in composition or language arts 
classes and only at accountable grade levels. 
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10. Evidence that students routinely use and can describe the steps in the writing process 

A. Students describe the steps in the writing process and assess their importance, when 
asked to do so 

High — All students interviewed name the essential steps of the writing process, often 
before they are asked to do so, and discuss cogently their importance in the students’ 
development as writers. 

Medium — Some of the students interviewed are highly aware of the steps of the writing 
process and can discuss their importance, while others must be asked fairly specific 
questions before they discuss them. Most students, however, exhibit some awareness of 
the various components of the writing process and have gone through them with at least 
some of their teachers. 

Low — Students fail to discuss some steps of the writing process at all (for instance, pre- 
writing or publication) and generally seem unaware that the various components of the 
process are important to help them develop as writers. 



B. Students are aware of criteria forjudging portfolios (and ways in which they 
learned these criteria) 

High — All the students interviewed report that each year one or more teachers reviews 
the criteria with them, going over and discussing benchmarks or particular pieces of 
writing that exemplify a particular performance level. Students are given written material 
reminding them of the criteria for each performance level, and the criteria are displayed or 
reviewed periodically as portfolios are being prepared. Students may report using the 
benchmarks on their own to assess their own work. 

Medium — All students report having at least one teacher during the accountability year 
go over criteria for the various performance levels, or the criteria may be displayed but not 
addressed consistently. Students feel free to ask their writing teachers if they need any 
additional information. 

Low — Students are not introduced to the criteria at all or are presented them in such 
simplified form that they are not really sure how their writing pieces are going to be 
judged. 

C. Use of computer to facilitate writing 

High — The school has sufficient computers so that students may have access to them at 
all stages of the writing process. Students have learned keyboarding skills at an early age, 
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have access to the Internet at school, and use computers to find information; they feel 
comfortable working on computers. Students have been encouraged to use the computer at 
all stages of the writing process and have learned from experience that this improves their 
writing, so most of the students interviewed choose to use the computer at all stages of the 
writing process, even if they do not have computers at home. At the time of our research, few 
schools had enough computers to rate high on this indicator. 

Medium — The school has sufficient computers so that students may have access to them 
at all stages of the writing process, but this is a recent development and not all teachers or 
students have grown accustomed to their use. Students who have computers at home still 
are more comfortable using the computer than those who do not and are more apt to use 
the computer at all stages of the writing process, but teachers are aware that students 
would benefit from greater use of the computer in writing. 

Low — The school either does not have enough computers for all students to use them at 
all stages of the writing process or they are not yet sufficiently accessible to students. 
Teachers are generally unaware of the advantages of using the computer at all stages of 
writing so do not encourage students to do so. Students who have computers at home are 
at an advantage in using computers for writing over students who do not. Computers at 
school are used mostly to produce final copy, and some students may not have access to 
them even for that purpose. Teachers may input copy for students rather than insisting 
they do it themselves, or students may use handwritten copy in their portfolios. 

D. Use of feedback to improve writing (conferencing with teachers, family, or other 
students) 

High — Teachers and students both report that students conference with peers as well as 
with teachers as they revise their writing pieces and that the feedback they receive from 
both sources is helpful. Students show no reluctance to share their writing with peers, 
even students they may not know well; they say that students at the school all know what 
sort of questions are helpful. Students share their writing with family members and friends 
outside of school and ask for their feedback (teaching them how to ask helpful questions, 
if necessary). Neither teachers nor students report problems that prevent students from 
receiving as much feedback as they need during the writing process. Students clearly feel 
ownership of their writing. 

Example: Teachers at one elementary school we visited taught primary 
students appropriate methods of conferencing and then made it clear 
that, since teachers and aides were not always available to conference 
with students, students who needed feedback were to find other 
students at about the same stage of composition and request a 
conference. Before they entered fourth grade, students had developed 
the habit of independently seeking feedback from other students as they 
worked on writing pieces. It was understood that they were supposed 
to do this, and they had the freedom to seek quiet places for the 
conferencing, including working quietly in the hallways. 
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Medium — Teachers and students report that students are expected to seek feedback 
from other students as well as teachers, but most students exhibit some reluctance to 
share their writing during intermediate stages. They may report avoiding conferencing 
with other students or seeking feedback only from close personal friends or students they 
trust to give good feedback. Teachers as well as students report that students seem to 
get more out of conferencing with teachers than out of conferencing with peers. They 
may have developed systems to increase the amount of time available for students to 
conference with adults (for instance, increasing the amount of time instructional aides can 
spend working one-on-one with students) but still feel there is a problem finding enough 
time for conferencing with students. 

Low — Even if teachers report that students do peer conferencing, the students report that 
they are seldom required to do this and that they can successfully avoid it most of the 
time. Teachers have little time to conference with students when they need help at 
various stages of the writing process, and they have not developed systems for increasing 
the amount of time instructors or aides are available to provide feedback to students. 
When they — or parents — do provide feedback, it tends to be primarily directive, so that 
students sometimes feel a piece of writing is no longer their work. 

11. Teacher writing 

Teachers who are not confident of their own ability as writers will presumably have difficulty 
helping students become proficient. Teachers who write with students appear to have a 
powerful effect on students' interest in writing and confidence that they can become 
proficient. 

A. Teachers write with or share their writing with students 

High — All teachers interviewed at the school can describe instances when they 
temporarily abandon the role of “teacher” to write with students. Often this will be 
journal writing, but to rate a “high” rating on this attribute, teachers should be able to 
describe occasions when they write at the same time as the students and share what 
they’ve written at the same time that students are sharing. 

Medium — Some of the teachers interviewed can describe instances when they write with 
students — but this is usually journal writing only. Some teachers report that they 
occasionally share with students a piece of writing that is similar to an assignment given 
the students. Students may also describe instances, but it is clear that such instances are 
occasional. 
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Example: The power of teachers sharing their own writing with 
students was illustrated during interviews with three ninth-grade 
students in a rural school district. When asked if they remembered 
any of their eighth-grade teachers sharing writing with them in 
class, all three recalled that their eighth-grade language arts 
teacher had read to them a narrative she had written about her 
granddaughter. Although interviews were conducted in May of 
their ninth-grade year, that memory was still vivid. 

Low — Neither teachers nor students describe any instances of teachers writing with 
students, other than writing on the chalkboard and requiring students to copy what they 
have written. 

B. Teachers introduce students to the many functions of writing in adult life 

High — Teachers report that they frequently talk with students about the ways they use 
writing in their everyday life and point out adult tasks and activities that require writing. 
From time to time, they invite adults in a variety of occupations to the classroom to 
discuss the role of writing in their work. Students are aware of a large number of ways in 
which writing is needed in adult life and can describe them to the interviewer. 



Medium — Some of the teachers interviewed discussed specific instances in which they 
brought to students’ attention the need for writing in one adult role or another. On rare 
occasions, an adult may have been invited to the school to enhance writing instruction by 
talking about the role of writing in their life and work. Students are aware that they will 
need to write as adults and can describe to the interviewer at least a few ways they 
anticipate writing as adults. 

Low — There is no evidence from either teachers or students that there is any discussion 
in the school of the role of writing in adult life and work. 

C. Evidence that teachers write frequently and for reasons other than teaching the 
writing process 

High — Most teachers, particularly language arts teachers, can easily give examples of 
personal writing, such as journals, letters, professional reports, and even poetry or fiction. 
It is clear from responses that most teachers regard writing as part of their personal and 
professional lives, not just as an instructional responsibility. 

Medium — Some teachers can easily give examples of personal writing. Others can 
provide one or two examples. However, the examples are few and mostly include 
obligatory writing such as reports required in their jobs. 
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Low — One or two teachers may give examples of personal writing. Most respond with 
minimal examples such as writing “to do” lists. 

12. Evidence of student choice of topics and formats for particular writing assignments 

High — All teachers interviewed at the school report that most assignments provide 
students with choice (appropriate to the subject area). Some assignments provide students 
with a format and a range of subjects within a particular prompt, but others allow students 
to choose the format to address a particular topic or allow for choice of both format and 
topic. Students are also encouraged to add to their writing folders pieces they have 
written not for school assignments but for real-world purposes outside school. Students 
agree with the teachers that they have many opportunities for choice of both format and 
topic in their writing and that most teachers give them opportunities for choice. 

Medium — Most teachers interviewed report giving assignments that specify format and 
the general topic area but allow students to make specific choices within the general area. 
Teachers may provide students with a list of possible topics or may brainstorm possible 
topics with the class and allow students to select from the list or choose a similar topic 
themselves. A few teachers report giving students a greater range of choices than this. 
Students agree with the teachers' reports. 

Low — All teachers interviewed report giving students a very narrow range of choices (for 
instance, specifying format and giving student three possible topics to choose among). 
Many writing assignments allow for no choice at all. Students agree that they exercise 
very little choice in their writing. 

13. Evidence of real-world writing 

A. Evidence of students having the opportunity to use writing for real-world purposes 

High — Teachers interviewed report that students have frequent opportunities to write to 
real audiences and that these communications are intended to have actual results (for 
instance, letters to the school board asking for specific improvements in the building or the 
lunch program, or pamphlets — later delivered — explaining the school's rules to an entering 
class). Students tell detailed stories about this type of writing, including the purpose for 
writing, what they wrote and why, and what the response was. 

Example: In one elementary school we visited, students were assigned 
to write to their parents prior to Christmas to ask for a particular present 
the child wanted very much but knew the parent might not be able to 
afford. The letters were mailed to the parents, and the children were able 
to tell the interviewer how his/her parents had responded to it. 
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Medium — Teachers report a variety of assignments to write to real audiences, but the 
letters are generally informational and not intended to produce particular results (for 
instance, letters to pen pals in other states or other countries). Teachers may also report a 
variety of assignments to write for imaginary audiences and may not differentiate sharply 
between the imaginary and the real audiences, understanding both as “real-world writing.” 
Students’ accounts agree with those of their teachers, except that students are normally 
quite aware which audiences are imaginary and which are real. 

Example: Offered as examples of real-world writing were letters to authors 
expressing appreciation for their writing. While such assignments are 
addressed to a real audience, they frequently have no purpose that is 
meaningful to the students, nor are they likely to produce a response 
beyond a form thank-you letter. 

Example: Some teachers asked students to write letters about current 
events that were not intended for mailing, or asked students to pretend they 
were characters in a novel or people living during a significant historical 
event writing letters to other fictional or historical characters. They 
sometimes offered such assignments as examples of real-world writing. 

Low — Neither teachers nor students mention writing for real-world purposes to actual 
audiences at all, and they do not describe such writing even when specifically asked about 
it. Teachers either do not understand the term “real-world writing” or describe writing 
for imaginary audiences when asked about it. 

B. Evidence that students are aware of the audiences for which they are writing and 
that the audiences are important to them 

High — Students discuss audience when asked to describe the writing process, and they 
differentiate clearly between imaginary or abstract audiences and actual audiences. They 
describe clearly what real audiences they have addressed in their writing, how they shape 
their writing to the audience, and why they think it's important to communicate with 
those audiences. They report that teachers encourage them to incorporate pieces they 
have written to real audiences for real-world purposes in their writing portfolios. 

Medium — Most students talk about audience only when prompted by the interviewer, 
but it is clear that they do write at least occasionally to audiences who read what they 
write. Students may or may not be able to describe the audiences’ reaction to what they 
wrote and may or may not think it is important to communicate with those audiences. 

Low — Neither teachers nor students report writing to real audiences as part of the 
writing program. Teachers may describe imaginary audiences in response to questions 
about real audiences. 
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14. Evidence of integration of writing instruction into general instruction 

A. Teachers using reading as a source of ideas and models for writing for a variety of 
purposes 

High — Teachers and students who are interviewed consistently talk about reading (in 
language arts or any of the other subject areas) as models for writing in that area and as a 
source of ideas for formats and topics. 

Medium — Some teachers and students who are interviewed can talk about particular 
ways in which classroom reading has served as a source of ideas for formats or topics or 
as a model of a type of writing. 

Low — Teachers and students discuss writing as totally separate from reading and do not 
draw the connection even when asked about reading as a source of ideas for writing. 

B. Language arts teachers teach mechanics in the context of writing 

High — Teachers and students who are interviewed report that mechanics are taught in 
the context of writing, primarily in mini-lessons as needed. Students speak of improved 
spelling and punctuation as part of the overall writing process. 

Medium — Teachers report that they use Daily Oral Language (DOL) as their primary 
method of teaching and reinforcing knowledge of mechanics. They report reinforcing 
these DOL lessons in the context of particular writing assignments or stressing in the 
DOL lessons mechanics they know will be important to upcoming writing assignments. 

Low — Teachers report teaching mechanics (spelling and grammar) as separate subjects. 
In teaching mechanics they follow the order of the textbook and do not make a special 
effort to teach mechanics with which students are having difficulty in their writing 
assignments. They may complain that time spent writing limits the amount of time for 
grammar instruction. 

15. Students receive consistent messages, consistent with Kentucky Writing Program 
principles, from all teachers 

High — Teachers interviewed at the school talk about the writing program as an 
integrated whole, even if teachers in different subject areas have different contributions to 
make to it. Their description of the objectives and progress of the writing program at the 
school reflects high-quality professional development and is essentially consistent from 
teacher to teacher and consistent with the way students describe the writing program. 
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Medium — A few teachers (usually language arts teachers) take the lead in implementing 
the writing program and have a clear vision among themselves, but the vision is not 
shared with other teachers; teachers in different subject areas or at different grade levels 
may have very different conceptions of the school’s writing program. Students report 
different understandings of the school's writing program, depending on the teachers they 
have, or their account of writing instruction differs from their teachers’ accounts. 

Example: In a high school we visited, two language arts teachers 
appeared to be largely responsible for the school's success in improving 
portfolio scores. Their students, when interviewed, were articulate about 
the various steps in the writing process and described challenging, 
interesting writing assignments, while students of other language arts 
teachers described only “cookie-cutter” assignments and appeared 
unaware of the writing process as a whole. 

Low — Students may receive consistent messages about writing, but the messages are that 
writing is not a school priority, and few or none of the faculty teach the writing process in 
the ways recommended by the Kentucky Writing Program. 

16. Communication by students of an awareness that becoming a good writer is a long-term 
process 

High — All the students interviewed at the school describe clearly how their writing has 
progressed over the past several years and make it clear that they expect to continue to 
improve. 

Medium — Students express an awareness that their writing has improved but do not 
project this awareness into the future, and some express disinterest in writing and no 
desire to continue to improve. 

Low — Students express little awareness of changes in their writing over time. 

17. Consistency among student/teacher/administrator perspectives 

This indicator requires a global judgment about the degree to which students, teachers, and 
administrators tell the same story in response to the same or similar questions. 

High — There are virtually no inconsistencies in what students, teachers, and 
administrators say; all describe the writing program in the same way, discussing a variety 
of writing assignments and telling revealing anecdotes about writing instruction. (It is 
possible, if not likely, to have a highly consistent depiction of writing in the school even 
when that depiction is negative.) 
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Medium — Students, teachers, and administrators describe the writing program in 
generally similar terms. There are inconsistencies, but for the most part they do not 
involve key indicators or add up to a grossly inconsistent story. If administrators have a 
different perception of the writing program, but teachers and students have consistent 
perceptions, the school is probably in the medium range on this indicator. 

Low — Descriptions of the writing program by students and teachers are so different they 
might be talking about different schools. Administrators may tell the same basic story as 
the teachers, or they may have a different perception of what is going on than the 
teachers. 



Statistical Findings 

Figure 1 displays the 36 indicators’ means scores in bar chart format for the 22 
continuously improving schools. The mean scores ranged from a low of 3.3 (Q10C, Use of 
computer to facilitate writing) to the high of 5.7 (Q7, Focus and intensity of the writing program). 
For the 36 indicators, there were 3 means in the 3s, 22 in the 4s, and 1.1 in the 5s. The pattern of 
means was for the first half of the figure to have more high means with a smaller range of scores 
than the second half of the figure. For example, for the first 18 indicators, there was just 1 mean 
in the 3s, 8 means in the 4s, and 9 means in the 5s; while for the second 18 indicators, there were 
2 means in the 3 s, 14 means in the 4s, and just 2 means in the 5 s. After the highest mean of 5.7 
(Q7), the next highest was 5.6 (Q4, School climate and communication), followed by 5.5 (Q1B, 
Administrative support — school level), followed next by 2 tied at 5.3 (Q2A, Professional 
development for writing leaders — building, and Q2E, Teacher training and participation in 
portfolio scoring). At the low end, after the lowest mean of 3.3 (Q10C, Use of computer to 
facilitate writing), the next lowest mean was 3.5 (Q5, Communication with families), followed 
next by 3.8 (Q13B, Evidence students are aware of audiences). All other indicator means for this 
continuously improving group of schools were 4.0 or above. 

Figure 2 displays the 36 indicators’ mean scores for the seven continuously declining 
schools. The mean scores range from a low of 1.2 (Q6, Use of community resources) to a high of 
3.7 (Q2E, Teacher training and participation in portfolio scoring). For the 36 indicators, there 
were 7 means in the Is, 22 means in the 2s, and 5 means in the 3s. As was the case for the 
previous grouping of schools (continuously improving), the pattern of means was for the first 18 
indicators to have more high means with a smaller range of scores that the second 1 8 indicators. 
For example, in Figure 2, all of the first 18 indicator mean scores were at or above 2.2, except for 
the aforementioned Q6 at 1.2. However, for the second 18 indicators, there were 8 means at 2.2 
or less, and the 2 highest means were 3. 1 (Q9C, Student confidence of reaching proficient) and 
3.0 (Q14B, Mechanics taught in context of writing). 

See Figure 3 for a display of the 36 indicators’ mean scores by the two groups of schools. 
This figure shows the comparison of the indicators’ mean scores for the continuously improving 
and the continuously declining schools. 
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Continuously Improving Group: Means for all Indicators 
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Figure 2: 

Continuously Declining Group: Means for all Indicators 



35 




ued|/\] 



CD 



Means for all Indicators by School Group 



36 



Table 1 displays the observed score differences, t-values, p-values, and significance levels 
of all 36 indicators between the continuously improving and the continuously declining schools. 
On the 7-point value scale, the observed score differences ranged from the low of 1 120 (Q1 1 A, 
Evidence teachers write and share with students) to the high of 3.149 (Q6, Use of community 
resources). The t-values ranged from the low of 1.799 (Q11A) to a very high 8.872 (QIC, 
Evidence teachers write frequently and independently). The probability values ranged from the 
largest of .083 (Q11A) to the smallest of .0001 (18 indicators). The last column shows only 1 
indicator (Q11A) to be not significant at the .05 level or less. Of the 35 indicators significant at 
the .05 level or less, 4 were significant at .05, 9 were significant at .01, 4 were significant at .001, 
and 18 were significant at .0001. Following the pattern of means in the figures above, the 
majority of indicators significant at the .0001 level were in the first page (Q1 A to Q9B). Further 
inspection of the descriptive statistics regarding indicator Q11A, the only non-significant one, 
revealed the reason for this result. For the 22 continuously improving schools, their mean on 
indicator Q1 1A (teachers write with students) was 3.977 with a standard deviation of 1.358, 
while the mean for the 7 continuously declining schools was 2.857 with a standard deviation of 
1.676. One continuously declining school had a score of 6.0 and another had a score of 4.0 on 
the 7-point scale. These two scores, out of the seven in this group, contributed greatly to the 
non-significance between the two groups on indicator Q11A. In the school in the declining group 
that scored at 6.0 on this indicator, a teacher reported writing with students, and then using that 
writing as the model for students to follow, rather than examining the models provided in the 
teacher’s writing handbook that illustrated benchmarks of writing competence. 

In addition to examining rating scores for high and low performing schools, an attempt 
was made to correlate 1995 school writing portfolio scores with factors such as school and class 
size, number of grades in the school, and socioeconomic status of students. No report is made 
of correlations between scores and geographic or ethnic descriptions due to the fact that most 
schools were in rural environs and most were mono-ethnic. Analysis found low correlations 
between portfolio scores and class size, school size, number of grades, or SES. None of the 
correlations was statistically significant. Distinctions between schools with improving and 
declining scores appeared to be based primarily on writing program practices and features 
identified by the indicators, rather than on school structures or social circumstances. 
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Table 1: Observed Score Differences, t-Values, p-Values, and Significance Levels 
of all Indicators Between the Continuously Improving and Continuously Declining Schools 
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CONCLUSIONS AND RECOMMENDATIONS 



At the completion of the research phase of this project, we draw four conclusions and 
make eight recommendations. The conclusions and recommendations concern both the identified 
indicators and the process by which those indicators were identified, as we believe both process 
and outcome have utility. 



Conclusions 

First, we conclude that it is possible to identify and validate writing program indicators 
that distinguish schools with continuously improving portfolio scores from those with 
continuously declining scores. Interestingly, indicators with statistically significant differences in 
mean scores between improving and declining schools are not limited to classroom practices of 
teachers charged with writing instruction, and the professional development of those teachers. 
They include indicators of the importance of writing in the school, and the involvement of non- 
language arts teachers in professional development on writing. They also include indicators for 
which no school, whether improving or declining, received high scores. These may point to 
program features that need increased professional development if schools are to continue 
improvements. 

Second, including students as informants about writing instruction proved invaluable in 
identifying common practices in the school. Teachers sometimes reported — and believed — that 
they were using a strategy effectively, yet their students seemed unaware of the strategy and could 
not exhibit knowledge or skills the strategy aimed to teach. Conversely, students sometimes 
demonstrated knowledge and skills beyond what teachers had anticipated. Student perceptions 
about the value of writing, its place in the school, and common writing practices helped to 
validate teacher and administrator perceptions. Consistency among the three role groups 
generally was a feature of a strong program. 

Third, the collaborative nature of the study provided unique benefits. Each group listened 
to and learned from the others. The researchers brought, in addition to research strategies, the 
neutral attitudes and cautions required for drawing well-founded conclusions from data. Writing 
consultants brought deep knowledge of the writing program from their experience as classroom 
teachers and as people charged with providing technical assistance to classroom teachers. They 
provided researchers with understandings of the complexities of school practice. Kentucky 
Writing Program directors brought awareness of the political and regulatory contexts of the 
program, as well as the ability to gain KDE support for research activities and school access. 

Each group had a vested interest in the research outcome and relied on the others to contribute to 
its success. At critical moments in the project, each relied on the credibility of the others to 
further the work. KDE staff relied on the dispassion of AEL as a research organization to 
support the credibility of preliminary findings presented to Kentucky legislators. AEL staff relied 
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on the writing consultants’ program understanding in validating school ratings on indicators. 

They also relied on Writing Program staff to lend authority to preliminary findings disseminated to 
school personnel, the KDE personnel, and the state legislature. 

Lastly, this collaboration is unique among Regional Educational Laboratories. More 
typically, laboratories — and other R&D institutions — carry out work that may be conceptualized 
collaboratively, but that is implemented independently. This project developed guidelines that 
may encourage similar collaborations in the future. Critical for successful collaborations of this 
sort is that (1) each agency staff already is accustomed to operating collaboratively, (2) there is a 
common vision of the outcome, (3) each has a vested interest in the outcome, and (4) each has 
strengths to contribute that are critical to the project’s success. All four conditions are important, 
but if the first does not exist, the others may not be enough to overcome its absence. 



Recommendations 

These findings should be disseminated widely within Kentucky and nationally. They have 
implications for school and district administrators, as well as classroom teachers. They have the 
potential to inform preservice education as well as in-service professional development. Since 
they concern the writing program as a whole, they also are of benefit to legislators concerned with 
the progress of school reform. Finally, because Kentucky’s writing program has many features in 
common with the National Writing Program used in other states with writing portfolio programs, 
they have import nationally. 

The instruments developed in this study are available for replication or adaptation in other 
R&D efforts. For example, it may be worthwhile to develop school-level indicators for other 
content areas such as social studies or science. Also, the writing program indicators should 
require little modification to apply to writing programs in other states that assess writing. 

A study using these developed protocols and processes should be undertaken in Kentucky 
schools in urban areas. Although the correlation between demographic factors and writing 
portfolio scores in this study was low, the relatively few urban schools in the sample makes 
conclusions based on these data tentative. 

Due to the small number of schools in the samples and the strict definitions governing the 
two school groups (continuously improving and continuously declining over the same four-year 
period), we also suggest that a similar study be conducted among schools excluded from this 
sample (i.e., schools with variable results). This population includes the majority of Kentucky’s 
schools. 

As a means of achieving the fourth recommendation, and to satisfy the need expressed by 
KDE staff for a process by which schools can assess their own writing programs, we recommend 
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that the instruments and processes used in the research phase be developed into a writing program 
self-study, then piloted and field-tested in schools not included in the research samples. 

Training should be designed that will enable people who provide technical assistance to 
schools to facilitate the writing program self-study recommended in the previous paragraph. 

Future studies of this kind should include student voice as an essential component. While 
student assessment scores offer an ultimate measure of program success, student perceptions 
allow insight into program elements and can help identify areas that can be improved to produce 
higher scores. 

The methods used in the research phase of this project are offered for the consideration of 
research institutions (institutes, centers, universities) as an example of high quality collaborative 
research and development that might be emulated. 
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APPENDIX 



Appendix A: 



Completed AEL SEDCAR Standards Checklist 



AEL SEDCAR Standards Checklist 



Study group report 0 Research report 

Field test report □ Impact study report 

Minigrant final report □ Other 

Implementation report 



The SEDCAR Standards were consulted and used as indicated in the table below (check or mark as appropriate): 



SEDCAR 
Standard Number 
and 

Descriptor 


The Standard was 
deemed applicable; and, 
to the extent feasible, 
was taken into account.* 


The Standard was 
deemed applicable; 
but ooukj not be taken 
into account. 


The Standard 
was not 
deemed 
applicable. 


Exception 
was taken 
to the 
Standard. 


1.1 


Creating an Infrastructure to Manage 
Data Collection Activities 


(St 


ate approved , no 


xxxx 

federal inv: 


)lvement) 


1.2 


Justifying Data Collection Activities 


xxxx 








1.3 


Fostering Commitment of all Participants 


xxxx 

(LEAs volunteered 


data) 






1.4 


Creating an Appropriate Management 
Process 


XXXX 








2.1 


Formulating and Refining Study Ques- 
tions 


xxxx 








2.2 


Choosing the Data Collection Methods 


xxxx 








2.3 


Developing a Sampling Plan 


xxxx 

(Plan was 50% of i 


dentified school* 


0 




2.4 


Assessing the Value of Obtainable Data 


xxxx 








2.5 


Transforming Study Question Concepts 
into Measures 


xxxx 








2.6 


Designing the Data Collection Instru- 
ment 


xxxx 

(Multiple instrume 


nts used) 






2.7 


Minimizing Total Study Error (Sampling 
and Nonsampling) 


XXXX 

(Mainly non-sampli 


ing errors) 






2.8 


Reviewing and Pretesting Data Collec- 
tion Instruments, Forms, and Procedures 


xxxx 

(Done in 7 pilot 1 


test schools) 




• 


2.9 


Preparing a Written Design 


XXXX 








3.1 


Preparing for Data Collection 


xxxx 








3.2 


Selecting and Training Data Collection 
Staff 


xxxx 

(Done in teams) 








3.3 


Ethical Treatment of Data Providers 


xxxx 








3.4 


Minimizing Burden and Nonresponse 


xxxx 








3.5 


Implementing Data Collection Quality 
Control Procedures 


xxxx 
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The Standards tor Educational Data □ 

Collection and Reporting (1991) Q 

were used in the development of p 

this (check one): 



SEDCAR 
Standard Number 
and 

Descriptor 


The Standard was 
deemed applicable; and, 
to the extent feasible, 
was taken into account* 


The Standard was 
deemed applicable; 
but could not be taken 
into account. 


The Standard 
was not 
deemed 
applicable. 


Exception 
was taken 
to the 
Standard. 


3.6 


Documenting Data Collections 


xxxx 








4.1 


Planning Systems Requirements 


xxxx 








4.2 


Designing Data Processing Systems 


xxxx 








4.3 


Developing Data Processing Systems 


xxxx 








4.4 


Testing Data Processing Systems 






xxxx 

(used stand< 


ird software 


4.5 


Planning for Data Preparation 


xxxx 








4.6 


Preparing Data for Processing and 
Analysis 


xxxx 








4.7 


Maintaining Programs and Data Files 


xxxx 








4.8 


Documenting Data Processing Activities 






xxxx 




4.9 


Evaluating Data Processing Systems 






xxxx 




5.1 


Preparing an Analysis Plan 


xxxx 








5.2 


Developing Analysis Variables 


xxxx 








5.3 


Applying Appropriate Weights 






xxxx 




5.4 


Estimating Sampling and Nonsampling 
Errors 






xxxx 




5.5 


Determining Statistical Significance 


xxxx 








6.1 


Presenting Findings 


xxxx 








6.2 


Reviewing the Report 


xxxx 








6.3 


Releasing Data 


xxxx 
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